Streaming media systems transmit media data, such as video and audio data, in data packets over wired and/or wireless networks. Due to constraints on available bandwidth, for example, some data packets transmitted over a network can experience delays along the way, perhaps arriving late at a destination node. Also, some data packets may be lost along the way.
The effects of late or lost data packets may be exacerbated for video data that are predictively encoded (compressed). Predictive encoding introduces dependencies in the encoded data that improve the amount of compression but can also result in error propagation in the event of data packet loss or late arrival.
Video frames are preferably received before their display (playback) times. With predictive encoding, the decoding of a frame of data may rely on the information in another frame, and therefore some frames need to be received earlier than their display time so that they can be used to decode other frames. For example, with MPEG (Moving Pictures Experts Group) encoding, a P-frame is predicted from a preceding P-frame or I-frame, and a B-frame is predicted from two P-frames or an I-frame and P-frame.
Encoded video frames that do not arrive or that arrive late at the decoder (e.g., a client or destination node) will not only miss their respective display deadlines, but they may also prevent a number of other, subsequent frames from being displayed properly. If, for example, a data packet constituting an I-frame or P-frame is lost, then it might not be possible to decode and display a subsequent P-frame or B-frame. In the case in which a P-frame cannot be decoded because a preceding I-frame or P-frame is absent, other subsequent frames dependent on that P-frame may not be decoded. Thus, a number of frames may be prevented from being properly decoded and displayed due to a single late or missing frame, depending on the particular coding dependencies of the late or missing frame. This can affect the overall quality of the display.
Video transcoding offers one solution that allows encoded data to be adapted to accommodate available bandwidth and packet losses. However, transcoding can be complex and thus may require significant computational effort.
Scalable encoding techniques have been developed, where encoded (compressed) data are prioritized and the prioritization provides a natural method for selecting which portion of compressed data to deliver within transmission constraints. In addition, conventional MPEG encoding also lends itself to a natural method of prioritizing data packets for delivery (e.g., I-frames are typically considered to be of higher priority than P-frames and B-frames). However, MPEG encoding generally treats all frames of a given frame type the same. For instance, all P-frames are treated as having equal importance), when in reality one P-frame may affect quality more than another. Recently, rate-distortion optimized packet scheduling has been proposed as a method for prioritizing data packets for delivery. While these techniques each have their advantages, they remain relatively complex and consequently computation-intensive.